The present invention relates to optimizing parameters for detection systems and, more particularly, to a method for finding parameter values that maximize the probability of detection for a selected number of probabilities of false alarm.
Today there is a tremendous amount of interest in systems that can detect radiological or nuclear threats. Many of these systems operate in extremely high throughput situations where delays caused by false alarms can have a significant negative impact. Thus, calculating the trade-off between detection rates and false alarm rates is critical for these detection systems' successful operation.
Receiver operating characteristic (ROC) curves have long been used to depict the tradeoff between detection and false alarm rates. The methodology was first developed in the field of signal detection. In recent years it has been used increasingly in machine learning and data mining applications. It follows that this methodology could be applied to threat detection systems. However many of these systems do not fit into the classic principles of statistical detection theory because they tend to lack tractable likelihood functions and have many parameters, which, in general, do not have a one-to-one correspondence with the detection classes. In short, currently there is no way to generate ROC curves for algorithms/systems that do not fit into the classic principles of statistical detection theory.
Given enough time and resources an estimation of all probability of detection, probability of false alarm (Pd, Pfa) pairs for an algorithm's parameter combinations at a reasonable granularity can be generated. These estimates can then be graphed on ROC scatter-plot. In general, each Pfa value will have several corresponding Pd values. In this case, only the highest Pd, and its associated parameter values, for a given Pfa are of interest. It is important to note that interpolating between points on this graph is liable to result in incorrect conclusions because it cannot be assumed that an interpolation between parameter values will generate the interpolated (Pd, Pfa) pair.
For algorithms with more than a handful of parameters performing this exhaustive search of the parameter space becomes virtually impossible. For example, to examine a relatively modest 4 values for each parameter of an algorithm with 10 parameters, results in 410 combinations of parameters for which Pd and Pfa values must be estimated. If each Pd and Pfa value requires a conservative 500 samples to compute, and each sample requires 20 seconds to process, the time to generate this very granular estimate of the (Pd, Pfa) pairs would take (410)(500)(20)≈1×1010 CPU-seconds, or about 330 CPU-years. Even after going through this exercise, it would still be questionable whether such a granular estimate would even be useful.
Thus, two options remain: a guided approach where domain experts and algorithm designers make their best guesses as to the sub-regions of this space to explore, or casting the problem into a global optimization task and use machine learning techniques to perform this optimization. While exploring the parameter space using domain experts and algorithm designers is useful, it can lead to the exclusion of productive portions of the parameter space.
What is needed then is a method that over comes these problems by empirically finding parameter values that maximize the probability of detection for a selected number of probabilities of false alarm.